Accelerating digital microscopy scans using empty/dirty area detection

ABSTRACT

A microscope including an illumination assembly, an image capture device and a processor can be configured to selectively identify regions of a sample including artifacts or empty space. By selectively identifying regions of the sample that have artifacts or empty space, the amount of time to generate an image of the sample and resources used to generate the image can be decreased substantially while providing high resolution for appropriate regions of the computational image. The processor can be configured to change the imaging process in response to regions of the sample that includes artifacts or empty space. The imaging process may include a higher resolution process to output higher resolution portions of the computational image for sample regions including valid sample material, and a lower resolution process to output lower resolution portions of the computational image for sample regions including valid sample material.

RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.16/875,721, filed May 15, 2020, now U.S. Pat. No. 11,409,095, issuedAug. 9, 2022, which is a bypass continuation of InternationalApplication No. PCT/IL2018/051253, filed Nov. 20, 2018, published as WO2019/097524, on May 23, 2019, and claims the benefit under 35 U.S.C. §119(e) of U.S. Provisional Application No. 62/588,658, filed Nov. 20,2017, the disclosures of which are incorporated, in their entirety, bythis reference.

BACKGROUND

The present disclosure relates generally to digital microscopy and/orcomputational microscopy and, more specifically, to systems and methodsfor accelerating digital microscopy by detecting sample artifacts andempty areas of the sample.

Microscopy is used in several applications and use cases to analyzesamples, such as a hospital, a lab or a clinic. A large volume of slidesmay need to be read at a microscope facility, and the throughput of suchsystems can be less than ideal. Commercial microscopes, such as wholeslide imaging (WSI) devices that are currently available, often comprisea single scanning microscope that relies primarily on accuratemechanical scanning and high-quality objective lenses. Recentlycomputational microscopy has been proposed as a way of improving theresolution of optical images. With computational microscopy, a pluralityof images of the sample can be obtained and processed to improveresolution and quality.

With these prior approaches to microscopy, the throughput may still belimited by the speed in which the scanning microscope can scan a singleslide, and by the computational time to generate the image in at leastsome instances. The prior approaches may less than ideally allocatemicroscope and processing resources and may sample and process moresample data than would be ideal. This can result in delays, resulting inless than ideal throughput.

Some facilities such as pathology labs and hospitals may scan severalmicroscope samples, and the throughput of the prior systems can be lessthan ideal. For example, some samples such as frozen samples, may needto be read quickly, while other samples may be less time sensitive. Inaddition, some samples may be read while a patient is in surgery todetermine how to treat the patient surgically. Also, the samplesobtained from tissue and other objects may contain artifacts or emptyregions that are not helpful in evaluating the sample. For example, withsome tissue samples such as needle biopsies and microtomes, the sampleon the microscope slide can be distributed unevenly.

The prior approaches to microscopy may scan more of the sample thanwould be ideal. For example, regions that contain artifacts or emptyspace may not be helpful in evaluating the sample. Examples of artifactsinclude particulate matter, dust, dirt, debris, and smudges. Theartifacts and empty space on the sample will generally not be helpful inanalyzing the sample. The prior approaches to microscopy can scan andprocess these regions with artifacts and empty space with resourcessimilar to other regions that contain useful sample material, resultingless than ideal throughput for the output images.

In light of the above, it would be desirable to have improved methodsand apparatus for increasing microscope imaging throughput at facility.Ideally, such improved microscope systems would overcome at least someof the aforementioned limitations of the prior approaches.

SUMMARY

As will be described in greater detail below, the instant disclosuredescribes various systems and methods for improving throughput ofmicroscopes such as computational microscopes. A microscope comprisingan illumination assembly, an image capture device and a processor can beconfigured to selectively identify regions of a sample comprisingartifacts or empty space. By selectively identifying regions of thesample that have artifacts or empty space, the amount of time togenerate an image of the sample and resources used to generate the imagecan be decreased substantially while providing high resolution forappropriate regions of the output image. The can be processor configuredto change the imaging process in response to regions of the sample thatcomprises artifacts or empty space. The imaging process may comprise ahigher resolution process to output higher resolution portions of thecomputational image for sample regions comprising valid sample material,and a lower resolution process to output lower resolution portions ofthe computational image for sample regions comprising valid samplematerial.

In an aspect, a microscope comprise an illumination assemble, an imagecapture device, and a processor. The illumination assembly can beoperable to illuminate a sample under observation of the microscope. Theimage capture device can be operable to capture an initial image set ofthe illuminated sample. The processor can be coupled to the imagecapture device and configured with instructions to identify an area ofthe sample that comprises at least one of artifact or empty space, andto determine a process for generating a computational image of thesample in response to identifying the area.

Features from any of the embodiments described herein may be used incombination with one another in accordance with the general principlesdescribed herein. These and other embodiments, features, and advantageswill be more fully understood upon reading the following detaileddescription in conjunction with the accompanying drawings and claims.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings illustrate a number of exemplary embodimentsand are a part of the specification. Together with the followingdescription, these drawings demonstrate and explain various principlesof the instant disclosure.

FIG. 1 is a diagram of an exemplary microscope, in accordance with someembodiments of the present disclosure.

FIG. 2 is a diagram of an exemplary sample under observation, inaccordance with some embodiments of the present disclosure.

FIG. 3 is a flowchart showing an exemplary process for acceleratingdigital microscopy scans using an empty and/or dirty area detection, inaccordance with some embodiments of the present disclosure.

FIG. 4 is a flowchart showing another exemplary process for acceleratingdigital microscopy scans using an empty and/or dirty area detection, inaccordance with some embodiments of the present disclosure.

Throughout the drawings, identical reference characters and descriptionsindicate similar, but not necessarily identical, elements. While theexemplary embodiments described herein are susceptible to variousmodifications and alternative forms, specific embodiments have beenshown by way of example in the drawings and will be described in detailherein. However, the exemplary embodiments described herein are notintended to be limited to the particular forms disclosed. Rather, theinstant disclosure covers all modifications, equivalents, andalternatives falling within the scope of the appended claims.

DETAILED DESCRIPTION

The presently disclosed microscope methods and apparatus disclosedherein can be used to measure many types of samples and generatecomputational images. The microscope can be configured to measureregions that comprise artifacts or are empty or dirty, for example. Insome embodiments, the images may comprise a computational image,although the present disclosure will find applications in other fields.By selectively identifying regions of the sample that are less useful,the speed of the imaging process can be increased. The methods andapparatus disclosed herein are well suited for use with one or morecomponents of prior systems. For example, the microscope methods andapparatus disclosed herein can be readily incorporated into priorsystems, for example with a software upgrade.

The artifact as described herein such as particulate matter, e.g. dustor debris, may be located from a focal plane of the microscope or withinit. The processor can be configured with instructions to determinewhether specific artifacts, portions or areas of the processed imagesoriginate from locations away from the focal plane, for example byexhibiting different shifts among an image set in response to adifferent illumination angle of the illumination beam.

FIG. 1 is a diagrammatic representation of a microscope 100 consistentwith the exemplary disclosed embodiments. The term “microscope” as usedherein generally refers to any device or instrument for magnifying anobject which is smaller than easily observable by the naked eye, i.e.,creating an image of an object for a user where the image is larger thanthe object. One type of microscope may be an “optical microscope” thatuses light in combination with an optical system for magnifying anobject. An optical microscope may be a simple microscope having one ormore magnifying lens. Another type of microscope may be a “computationalmicroscope” that comprises an image sensor and image-processingalgorithms to enhance or magnify the object's size or other properties.Enhancements may include resolution enhancement, quality improvement(e.g., aberration correction, computational refocusing, contrastenhancement, distortion correction, color enhancement, registration,removing certain elements of the data, etc.). The computationalmicroscope may be a dedicated device or created by incorporatingsoftware and/or hardware with an existing optical microscope to producehigh-resolution digital images. As shown in FIG. 1 , microscope 100comprises an image capture device 102, a focus actuator 104, acontroller 106 connected to memory 108, an illumination assembly 110,and a user interface 112. An example usage of microscope 100 may becapturing images of a sample 114 mounted on a stage 116 located withinthe field-of-view (FOV) of image capture device 102, processing thecaptured images, and presenting on user interface 112 a magnified imageof sample 114.

Image capture device 102 may be used to capture images of sample 114. Inthis specification, the term “image capture device” as used hereingenerally refers to a device that records the optical signals entering alens as an image or a sequence of images. The optical signals may be inthe near-infrared, infrared, visible, and ultraviolet spectrums.Examples of an image capture device comprise a CCD camera, a CMOScamera, a photo sensor array, a video camera, a mobile phone equippedwith a camera, a webcam, a preview camera, a microscope objective anddetector, etc. Some embodiments may comprise only a single image capturedevice 102, while other embodiments may comprise two, three, or evenfour or more image capture devices 102. In some embodiments, imagecapture device 102 may be configured to capture images in a definedfield-of-view (FOV). Also, when microscope 100 comprises several imagecapture devices 102, image capture devices 102 may have overlap areas intheir respective FOVs. Image capture device 102 may have one or moreimage sensors (not shown in FIG. 1 ) for capturing image data of sample114. In other embodiments, image capture device 102 may be configured tocapture images at an image resolution higher than VGA, higher than 1Megapixel, higher than 2 Megapixels, higher than 5 Megapixels, 10Megapixels, higher than 12 Megapixels, higher than 15 Megapixels, orhigher than 20 Megapixels. In addition, image capture device 102 mayalso be configured to have a pixel size smaller than 15 micrometers,smaller than 10 micrometers, smaller than 5 micrometers, smaller than 3micrometers, or smaller than 1.6 micrometer.

In some embodiments, microscope 100 comprises focus actuator 104. Theterm “focus actuator” as used herein generally refers to any devicecapable of converting input signals into physical motion or changing rayconvergence for adjusting the relative distance between sample 114 andimage capture device 102. Various focus actuators may be used,including, for example, linear motors, electrostrictive actuators,electrostatic motors, capacitive motors, voice coil actuators,magnetostrictive actuators, liquid lenses, etc. In some embodiments,focus actuator 104 may comprise an analog position feedback sensorand/or a digital position feedback element. Focus actuator 104 isconfigured to receive instructions from controller 106 in order to makelight beams converge to form a clear and sharply defined image of sample114. In the example illustrated in FIG. 1 , focus actuator 104 may beconfigured to adjust the distance by moving image capture device 102.

However, in other embodiments, focus actuator 104 may be configured toadjust the distance by moving stage 116, or by moving both image capturedevice 102 and stage 116. Microscope 100 may also comprise controller106 for controlling the operation of microscope 100 according to thedisclosed embodiments. Controller 106 may comprise various types ofdevices for performing logic operations on one or more inputs of imagedata and other data according to stored or accessible softwareinstructions providing desired functionality. For example, controller106 may comprise a central processing unit (CPU), support circuits,digital signal processors, integrated circuits, cache memory, or anyother types of devices for image processing and analysis such as graphicprocessing units (GPUs). The CPU may comprise any number ofmicrocontrollers or microprocessors configured to process the imageryfrom the image sensors. For example, the CPU may comprise any type ofsingle- or multi-core processor, mobile device microcontroller, etc.Various processors may be used, including, for example, processorsavailable from manufacturers such as Intel®, AMD®, etc. and may comprisevarious architectures (e.g., x86 processor, ARM®, etc.). The supportcircuits may be any number of circuits generally well known in the art,including cache, power supply, clock and input-output circuits.Controller 106 may be at a remote location, such as a computing devicecommunicatively coupled to microscope 100.

In some embodiments, controller 106 may be associated with memory 108used for storing software that, when executed by controller 106,controls the operation of microscope 100. In addition, memory 108 mayalso store electronic data associated with operation of microscope 100such as, for example, captured or generated images of sample 114. In oneinstance, memory 108 may be integrated into the controller 106. Inanother instance, memory 108 may be separated from the controller 106.

Specifically, memory 108 may refer to multiple structures orcomputer-readable storage mediums located at controller 106 or at aremote location, such as a cloud server. Memory 108 may comprise anynumber of random access memories, read only memories, flash memories,disk drives, optical storage, tape storage, removable storage and othertypes of storage.

Microscope 100 may comprise illumination assembly 110. The term“illumination assembly” as used herein generally refers to any device orsystem capable of projecting light to illuminate sample 114.

Illumination assembly 110 may comprise any number of light sources, suchas light emitting diodes (LEDs), LED array, lasers, and lamps configuredto emit light, such as a halogen lamp, an incandescent lamp, or a sodiumlamp. In one embodiment, illumination assembly 110 may comprise only asingle light source. Alternatively, illumination assembly 110 maycomprise four, sixteen, or even more than a hundred light sourcesorganized in an array or a matrix. In some embodiments, illuminationassembly 110 may use one or more light sources located at a surfaceparallel to illuminate sample 114. In other embodiments, illuminationassembly 110 may use one or more light sources located at a surfaceperpendicular or at an angle to sample 114. Illumination assembly 110may comprise other optical elements, such as lenses, mirrors, diffusers,active or passive phase elements, intensity elements, etc.

In addition, illumination assembly 110 may be configured to illuminatesample 114 in a series of different illumination conditions. In oneexample, illumination assembly 110 may comprise a plurality of lightsources arranged in different illumination angles, such as atwo-dimensional arrangement of light sources. In this case, thedifferent illumination conditions may comprise different illuminationangles. For example, FIG. 1 depicts a beam 118 projected from a firstillumination angle α1, and a beam 120 projected from a secondillumination angle α2. In some embodiments, first illumination angle α1and second illumination angle α2 may have the same value but oppositesign. In other embodiments, first illumination angle α1 may be separatedfrom second illumination angle α2. However, both angles originate frompoints within the acceptance angle of the optics. In another example,illumination assembly 110 may comprise a plurality of light sourcesconfigured to emit light in different wavelengths. In this case, thedifferent illumination conditions may comprise different wavelengths. Inyet another example, illumination assembly 110 may configured to use anumber of light sources at predetermined times. In this case, thedifferent illumination conditions may comprise different illuminationpatterns. Accordingly and consistent with the present disclosure, thedifferent illumination conditions may be selected from a groupincluding: different durations, different intensities, differentpositions, different illumination angles, different illuminationpatterns, different wavelengths, or any combination thereof.

Consistent with disclosed embodiments, microscope 100 may comprise, beconnected with, or in communication with (e.g., over a network, viadedicated connection (e.g., HDMI, VGA, RGB, Coaxial) or wirelessly,e.g., via Bluetooth or WiFi) user interface 112. The term “userinterface” as used herein generally refers to any device suitable forpresenting a magnified image of sample 114 or any device suitable forreceiving inputs from one or more users of microscope 100. FIG. 1illustrates two examples of user interface 112. The first example is asmartphone or a tablet wirelessly communicating with controller 106 overa Bluetooth, cellular connection or a Wi-Fi connection, directly orthrough a remote server. The second example is a PC display or monitorphysically connected to controller 106. In some embodiments, userinterface 112 may comprise user output devices, including, for example,a display, tactile device, speaker, etc. In other embodiments, userinterface 112 may comprise user input devices, including, for example, atouchscreen, microphone, keyboard, pointer devices, cameras, knobs,buttons, etc. With such input devices, a user may be able to provideinformation inputs or commands to microscope 100 by typing instructionsor information, providing voice commands, selecting menu options on ascreen using buttons, pointers, or eye-tracking capabilities, or throughany other suitable techniques for communicating information tomicroscope 100. User interface 112 may be connected (physically orwirelessly) with one or more processing devices, such as controller 106,to provide and receive information to or from a user and process thatinformation. In some embodiments, such processing devices may executeinstructions for responding to keyboard entries or menu selections,recognizing and interpreting touches and/or gestures made on atouchscreen, recognizing and tracking eye movements, receiving andinterpreting voice commands, etc.

Microscope 100 may also comprise or be connected to stage 116. Stage 116comprises any horizontal rigid surface where sample 114 may be mountedfor examination. Stage 116 may comprise a mechanical connector forretaining a slide containing sample 114 in a fixed position. Themechanical connector may use one or more of the following: a mount, anattaching member, a holding arm, a clamp, a clip, an adjustable frame, alocking mechanism, a spring or any combination thereof. In someembodiments, stage 116 may comprise a translucent portion or an openingfor allowing light to illuminate sample 114. For example, lighttransmitted from illumination assembly 110 may pass through sample 114and towards image capture device 102. In some embodiments, stage 116and/or sample 114 may be moved using motors or manual controls in the XYplane to enable imaging of multiple areas of the sample.

FIG. 2 is a diagram of an exemplary sample 200 comprising a plurality ofareas or regions, in which some of the areas comprise valid sampleportions and other areas comprise one or more of artifacts or emptyspace. The sample 200 may comprise a plurality of areas that can beevaluated to determine whether the areas comprises valid sample data,artifacts or empty space. The sample 200 may comprise a valid portion210 generally extending across a plurality of areas. In someembodiments, the valid portion 210 of the sample 200 comprises an edge213, which may extend through a plurality of imaged areas of the sample.The plurality of areas may comprise a first area 201. The first area maycomprise a valid portion of the sample and an unreliable portion of thesample comprising artifact or empty space. For example, the plurality ofareas may comprise an empty area 202, corresponding to voids in thevalid portion. The empty area 202 can be identified by the microscopecomprising the processor as described herein, and the processor canchange the imaging process in response to the identified area. Theplurality of areas may comprise a third area 203 comprising artifacts asdescribed herein such as particulate matter, for example. The sample 200may comprise a fourth area 204 comprising a portion of valid sample overat least a portion of the region. Valid portion 210 may comprisematerial such as biological material that is of interest and can be usedto generate high resolution images. The microscope 100 may be configuredto change the imaging process for some areas of the sample and may onlypartially process areas 202 and 203, for example. This has the advantageof reducing processing by microscope 100 and expediting computational orother imaging of sample 200. Although reference is made to a pluralityof images of the sample, a single image can be processed in accordancewith the present disclosure. Also, the image set used to generate thecomputational microscopy images as disclosed herein may comprise asingle image, or a plurality of images, for example. The regions of thesample comprising at least a portion of the valid sample, such asregions 201 and 204 can be imaged with higher resolution.

The processor can be configured with instructions to generate thecomputational image in accordance with regions corresponding to theidentified areas of the sample comprising valid sample data, artifactsor empty space. For example, the computational image may comprise ahigher spatial resolving power at regions corresponding to valid portion210, and lower spatial resolving power at regions outside valid portion210. For example, regions of the computational image corresponding toregion 202 and 203 may comprise lower spatial resolving power. In someembodiments, the lower resolution image comprises the same number ofpixel density as other regions, which can be generated by emptymagnification or interpolation. These approaches can provide pixelresolution enhancement without increasing spatial resolving power of theportion image, and the processor can be configured with appropriateinstructions.

FIG. 3 is a flowchart showing an exemplary process 300 for acceleratingdigital microscopy scans using an empty and/or dirty area detection. Inthis embodiment, microscope 100 illuminates a sample, such as sample 200of FIG. 2 , under observation, at step 302. Image capture device 102 ofmicroscope 100 captures an initial image set of the illuminated sample(e.g., one or more images), at step 304. The order of steps 302 and 304can be changed, and these steps can be performed in parallel or in arepeating fashion until the image set is captured. Also, other steps canbe used, such as generating a composite image or partial computationalimage of the sample.

A processor of microscope 100, such as controller 106, tests whether anarea of at least one image has artifact and/or empty space, at step 306.For example, the processor may scan sample 200 and identify areas 202and/or 203 as empty or having artifact, e.g., having no discernibleviewing interest. Alternatively or in combination, information from aplurality of images, such as a computational image or appearance ofartifacts in several images, can be used with step 306. In someembodiments, the processor is configured with instructions to search forvalid data, and determine that the area is empty or contains artifactsif the amount of valid data found in the area is under a thresholdamount or does not meet a defined criterion for valid data. For example,the processor can be configured with instructions to identify areascomprising artifact or empty space, or instructions to identify validdata, and combinations thereof. The processor can be configured withinstructions to separately test for each of artifacts or empty space,either separately or in combination.

The area or regions of the sample under test can be provided in step 306in many ways and in some embodiments without steps 302 and 304. Forexample, the area under test can be provided to the processor andprocessed to determine whether the area has valid data, artifact orempty space. Any source of image data as described herein can be used toperform the test at step 306.

At step 308 the processor determines whether an area has artifact and/orempty space.

The test for artifact and/or empty data can be configured in many wayseither alternatively or in combination. For example, the test can beperformed on a composite image or a computational image or fromanalyzing similarities or differences between images (e.g a portionwhich may look like valid data in a single image or some of the imagesmay not be present in other images and may be interpreted as anartifact). Also, the test can be performed on any one or more images ofthe image set, or any portion of the one or more images of the imageset, or other image data for example. This test can be configured todetermine when there is valid data, artifacts or empty space. Also, thetesting can be configured to provide statistical data such as aprobability that the area or region comprises, valid data, artifacts, orempty space, as described herein. The probability can be used to suggestthat the tested area or region comprises valid data, artifacts, or emptyspace. Also, additional or alternative metrics to probability can beused, such as analysis of spatial frequencies, to test the area orregion. In this regard, the test can determine whether the areapotentially has artifact. At step 308, the “yes” or “no” test can beperformed based on a statistical or other analysis to alter the processas described herein.

If the area comprises artifact or empty space, the processor may directmicroscope 100 to skip the area and/or only partially process the areato reduce computational imaging at step 310. This process can beemployed during the acquisition of images with the image capture device,or later during the image reconstruction process, and combinationsthereof. If the area does not comprise artifact or empty space, the areacan be process with high resolution at step 314. Although a highresolution process is shown, other processes can be performed eitheralternatively or in combination. The process may improve other aspectsof the image related to image quality, such as quality improvement,aberration correction, computational refocusing, contrast enhancement,distortion correction, color enhancement, registration, removingidentified elements of the data. In some embodiments, the removedidentified elements of the data comprise one or more of artifact, dustor empty space.

The processor may also flag the areas having artifact and/or empty spacefor subsequent imaging at step 312. For example, as microscope 100initiates more in-depth computational imaging of the sample, those areasof the sample that are of little or no interest may be flagged such thatmicroscope 100 forgoes any additional processing of those areas.

At a step 316, the imaging process moves to the next area of the sample.

FIG. 4 is a flowchart showing another exemplary process 400 foraccelerating digital microscopy scans using artifact and/or dirty areadetection. In this embodiment, the processor of microscope 100determines whether microscope 100 is in a full resolution mode or apartial/empty (PE) mode, at step 402. For example, if microscope 100 wasscanning in region 201 of sample 200 of FIG. 2 , microscope 100 wouldtypically employ full resolution imaging. Accordingly, microscope 100may remain in the full resolution mode and assume that data in a nextarea is of interest. In this regard, the processor may perform animaging process on that subsequent area at step 412.

In some embodiments the microscope processor can be configured with onemode, and the mode selection described herein is optional. e.g. only onemode, which tests for empty or dirty areas while processing fullresolution images, or PE mode without switching to full resolution mode.

The processor may then computationally process the image at step 414 togenerate a computational image. Generally, a computational image is animage where at least a part of the image was created using acomputational process. For example, a computational process may includeresolution enhancement and/or quality improvement, such as, aberrationcorrection, computational refocusing, contrast enhancement, distortioncorrection, color enhancement, registration, and/or removing certainelements of the data such as debris, dust, and/or empty space. Someprocesses that may be applied to areas that are empty or dirty (e.g.,areas 202 and 203 of FIG. 2 ) include the computational removal of dirtand/or artifacts related to it, full resolution or quality enhancement,partial resolution or quality enhancement, and empty resolutionenhancement (e.g., by increasing the pixel count without improving theoptical resolving power, also known as interpolation).

In this regard, the processor of microscope 100 may perform additionaland/or computational imaging processes if the area being observed isalso being tested, at step 416. Step 416 is an optional step, and candepend on other aspects of the work flow and process 400 and otherprocesses and methods as described herein. For example, step 416 can beperformed when the microscope has identified an area as having relevantdata and partially constructed the image as part of step 404.Alternatively, step 416 can be skipped, for example when the processcomprises the full resolution mode, and process 414 has generated thecomputational image.

Thereafter, the processor may adjust the testing criteria used fortesting particular area, at step 418. This may allow the processor tochange modes from full resolution mode to PE mode. For example, as theprocessor is testing an area of the image, the processor may deem thearea as either empty or dirty before proceeding to a subsequent area. Asa subsequent area may likely be empty or dirty as well, the processormay switch to the PE mode at step 420 for the subsequent area.Conversely, if the processor encounters valid data (e.g., a portion ofthe image occupied by region 201 of FIG. 2 ), the processor may bedetermined that a subsequent region may also comprise valid data. Inthis regard, the processor may set microscope 100 to operate in the fullresolution mode at step 420. Although reference is made to the fullresolution mode, the full resolution mode may comprise one or moreadditional or alternative computational processes generally related tothe quality of the image, such as aberration correction, etc.

Microscope 100 may move on to the next area, at step 422, and returnedto step 402. If there is no reason to change the testing criteria (e.g.,because the current mode of microscope 100 is likely to be used in asubsequent area), the processor may direct microscope 100 to simply moveon to the next area, at step 422.

Returning to step 402, if the processor determines that microscope 100is operating in the PE mode, the processor may direct microscope 100 toperform an imaging process, at step 404. For example, the processor mayform a partial imaging of an area and then test whether the area hasdebris and/or empty space, at step 406. In some embodiments, the partialor full imaging partial process may be limited to the computationalprocess, while other processes such as image illumination andacquisition continue. This partial imaging may reduce the computationalcomplexity and thus reduce the number of computations used in theimaging. Then, the processor may they determine whether the areaincludes relevant data or not, at step 408.

If the area does include relevant data, the processor may directmicroscope 100 to operate in full resolution mode and generate acomputational process image, at step 414. Otherwise, the processor maypartially process the image or even skip over the entire area, at step410.

Any of the steps of method 400 can be combined with any method stepcorresponding to a block of workflow 300 as described herein. Althoughworkflow 300 and method 400 are described as a sequence of steps, insome embodiments various concurrent iterations may result in steps beingstalled, omitted, repeated, and/or performed in different order. Thesteps disclosed herein are optional, e.g. steps 302 and 304, and can beperformed in any order.

As detailed above, the computing devices and systems described and/orillustrated herein broadly represent any type or form of computingdevice or system capable of executing computer-readable instructions,such as those contained within the modules described herein. In theirmost basic configuration, these computing device(s) may each comprise atleast one memory device and at least one physical processor.

The term “memory” or “memory device,” as used herein, generallyrepresents any type or form of volatile or non-volatile storage deviceor medium capable of storing data and/or computer-readable instructions.In one example, a memory device may store, load, and/or maintain one ormore of the modules described herein. Examples of memory devicescomprise, without limitation, Random Access Memory (RAM), Read OnlyMemory (ROM), flash memory, Hard Disk Drives (HDDs), Solid-State Drives(SSDs), optical disk drives, caches, variations or combinations of oneor more of the same, or any other suitable storage memory.

In addition, the term “processor” or “physical processor,” as usedherein, generally refers to any type or form of hardware-implementedprocessing unit capable of interpreting and/or executingcomputer-readable instructions. In one example, a physical processor mayaccess and/or modify one or more modules stored in the above-describedmemory device. Examples of physical processors comprise, withoutlimitation, microprocessors, microcontrollers, Central Processing Units(CPUs), Field-Programmable Gate Arrays (FPGAs) that implement softcoreprocessors, Application-Specific Integrated Circuits (ASICs), portionsof one or more of the same, variations or combinations of one or more ofthe same, or any other suitable physical processor.

Although illustrated as separate elements, the method steps describedand/or illustrated herein may represent portions of a singleapplication. In addition, in some embodiments one or more of these stepsmay represent or correspond to one or more software applications orprograms that, when executed by a computing device, may cause thecomputing device to perform one or more tasks, such as the method step.

In addition, one or more of the devices described herein may transformdata, physical devices, and/or representations of physical devices fromone form to another. For example, one or more of the devices recitedherein may receive image data of a sample to be transformed, transformthe image data, output a result of the transformation to determine a 3Dprocess, use the result of the transformation to perform the 3D process,and store the result of the transformation to produce an output image ofthe sample. Additionally or alternatively, one or more of the modulesrecited herein may transform a processor, volatile memory, non-volatilememory, and/or any other portion of a physical computing device from oneform of computing device to another form of computing device byexecuting on the computing device, storing data on the computing device,and/or otherwise interacting with the computing device.

The term “computer-readable medium,” as used herein, generally refers toany form of device, carrier, or medium capable of storing or carryingcomputer-readable instructions. Examples of computer-readable mediacomprise, without limitation, transmission-type media, such as carrierwaves, and non-transitory-type media, such as magnetic-storage media(e.g., hard disk drives, tape drives, and floppy disks), optical-storagemedia (e.g., Compact Disks (CDs), Digital Video Disks (DVDs), andBLU-RAY disks), electronic-storage media (e.g., solid-state drives andflash media), and other distribution systems.

The process parameters and sequence of the steps described and/orillustrated herein are given by way of example only and can be varied asdesired. For example, while the steps illustrated and/or describedherein may be shown or discussed in a particular order, these steps donot necessarily need to be performed in the order illustrated ordiscussed. The various exemplary methods described and/or illustratedherein may also omit one or more of the steps described or illustratedherein or comprise additional steps in addition to those disclosed.

The processor as disclosed herein can be configured to perform any oneor more steps of a method as disclosed herein.

The preceding description has been provided to enable others skilled inthe art to best utilize various aspects of the exemplary embodimentsdisclosed herein. This exemplary description is not intended to beexhaustive or to be limited to any precise form disclosed. Manymodifications and variations are possible without departing from thespirit and scope of the instant disclosure. The embodiments disclosedherein should be considered in all respects illustrative and notrestrictive. Reference should be made to the appended claims and theirequivalents in determining the scope of the instant disclosure.

Unless otherwise noted, the terms “connected to” and “coupled to” (andtheir derivatives), as used in the specification and claims, are to beconstrued as permitting both direct and indirect (i.e., via otherelements or components) connection. In addition, the terms “a” or “an,”as used in the specification and claims, are to be construed as meaning“at least one of” Finally, for ease of use, the terms “including” and“having” (and their derivatives), as used in the specification andclaims, are interchangeable with and have the same meaning as the word“comprising.” Also, as used herein the term “multiple” encompasses a“plurality” and refers to two or more.

This disclosure also includes the following numbered clauses

Clause 1. A microscope, comprising:

-   -   an illumination assembly operable to illuminate a sample under        observation of the microscope;    -   an image capture device operable to capture an initial image set        of the illuminated sample; and    -   a processor coupled to the image capture device and configured        with instructions to identify an area of the sample that        comprises at least one of artifact or empty space, and to        determine a process for generating a computational image of the        sample in response to identifying the area.

Clause 2. The microscope of clause 1, wherein:

-   -   the processor is configured to output to a display portions of        the computational image of the sample that comprise the at least        one of artifact or empty space at a higher rate than other        portions of the computational image of the sample that do not        comprise the at least one of artifact or empty space.

Clause 3. The microscope according to any of clauses 1 to 2, wherein:

-   -   the processor is configured with instructions to operate in a        first mode in response to identifying the area of the sample        comprising the at least one of artifact or empty space and in a        second mode in response to identifying an area of the sample        that does not comprise the at least one of artifact or empty        space; and    -   in the first mode and the second mode the processor is        configured to output portions of the computational image with a        first spatial resolving power and second spatial resolving        power, respectively, the first spatial resolving power less than        the second spatial resolving power.

Clause 4. The microscope according to any of clauses 1 to 3, wherein:

-   -   the image set comprises a plurality of regions;    -   the processor is configured with instructions to run a first        process to increase spatial resolving power of each of the        plurality of regions and to run a second process to determine        whether each of the plurality of regions comprises the at least        one of artifact or empty space;    -   optionally the second process is initiated before the first        process has completed; and    -   optionally the first process is stopped in response to the        second process identifying one or more of the plurality of        regions comprising the at least one of artifact or empty space.

Clause 5. The microscope according to any of clauses 1 to 4, wherein:

-   -   the illumination assembly is configured to illuminate the sample        at a plurality of illumination conditions and the image capture        device generates a plurality of images, each of the plurality of        images corresponding to a different illumination condition; and    -   the computational image is generated from the plurality of        images;    -   the at least one of artifact or empty space is identified from        the computational image or one or more of the plurality of        images; and    -   the plurality of illumination conditions comprises at least one        of an illumination angle, an illumination wavelength, or an        illumination pattern.

Clause 6. The microscope according to any of clauses 1 to 5, wherein:

-   -   the processor comprises instructions to detect an area of the        sample that comprises at least one of artifact or empty space,        by using one or more of: a classifier, a detail in a darkfield        image, a brightness, a color distribution, a cross correlation,        a Fourier analysis, an entropy, a feature detection, a shift of        details, a sparsity, an edge detection, a three-dimensional        reconstruction, a depth detection, or a light field.

Clause 7. The microscope according to any of clauses 1 to 6, furthercomprising:

-   -   an input comprising pixels from an image set of the sample, and        an output comprising the at least one of artifact or empty        space.

Clause 8. The microscope according to any of clauses 1 to 7, wherein:

-   -   the computational image comprises a two-dimensional image.

Clause 9. The microscope according to any of clauses 1 to 8, wherein:

-   -   the computational image comprises one or more of a        three-dimensional image or a three-dimensional model of the        sample.

Clause 10. The microscope according to any of clauses 1 to 9, furthercomprising:

-   -   another image capture device with a higher resolving power than        said image capture device, the other image capture device being        used to image the sample at the higher resolving power than said        image capture device.

Clause 11. The microscope according to any of clauses 1 to 10, wherein:

-   -   the initial image set image comprises a preview image; and    -   the processor is configured with instructions to determine a        boundary of the sample in the preview image, and to generate        another image of the sample, wherein the other image comprises a        higher resolving power within the boundary of the of the sample        and a lower resolving power in the identified area.

Clause 12. The microscope according to any of clauses 1 to 11, wherein:

-   -   the higher resolving power corresponds to smaller spatial        features being resolved than the lower resolving power.

Clause 13. The microscope according to any of clauses 1 to 12, wherein:

-   -   the computational image comprises a digital image; and    -   the processor is configured with instructions to remove data        pertaining to the identified area to change a resolving power of        the identified area.

Clause 14. The microscope according to any of clauses 1 to 13, wherein:

-   -   the processor is configured with instructions to identify the        area as comprising the at least one of artifact or empty space        in response to a shift in position of portions of the image set        or computational image in response to an off-axis illumination        of the sample.

Clause 15. The microscope according to any of clauses 1 to 14, wherein:

-   -   the microscope comprises a computational microscope configured        to generate an image from a plurality of images of the sample        with different illumination conditions; and    -   the illumination different conditions comprise at least one of        different illumination angles, different illumination        wavelengths, different illumination patterns, different        illumination durations, different illumination intensities, or        different illumination positions.

Clause 16. The microscope according to any of clauses 1 to 15, furthercomprising:

-   -   a translation stage to move the sample to a plurality of        locations in order to scan the sample and generate a plurality        of images, wherein the microscope is configured to scan the        sample faster in response to the area that comprises at least        one of artifact or empty space as compared to the scan speed of        an area of the sample that comprises a valid portion of the        sample.

Clause 17. The microscope according to any of clauses 1 to 16, wherein:

-   -   the image capture device captures the initial image set of the        illuminated sample using a plurality of illumination conditions        for illuminating the sample; and    -   the plurality of illumination conditions comprises at least one        of different illumination angles, different illumination        wavelengths, different illumination patterns, different        illumination durations, different illumination intensities, or        different illumination positions.

Clause 18. The microscope according to any of clauses 1 to 17, wherein:

-   -   a higher resolving power portion of the image comprises resolved        details with a smaller spatial distance than resolved details of        a lower resolving power portion of the image.

Clause 19. The microscope according to any of clauses 1 to 18, wherein:

-   -   the processor is configured with instructions to reduce        computational complexity when processing in the area of the        sample that comprises the at least one of artifact or empty        space to reduce computation time.

Clause 20. The microscope according to any of clauses 1 to 19, wherein:

-   -   the processor is configured with instructions to use fewer        illumination conditions when processing in the area of the        sample that comprises the at least one of artifact or empty        space as compared to an area of the sample that does not        comprise the at least one of artifact or empty space.

Clause 21. The microscope according to any of clauses 1 to 20, wherein:

-   -   the computational image comprises a first area comprising a        first resolving power and a second area comprising a second        resolving power less than the first resolving power and        optionally wherein the second area corresponds to the at least        one of artifacts or empty space of the sample.

Clause 22. The microscope according to any of clauses 1 to 21, wherein:

-   -   a low resolving power portion of the computational image        comprises an image with a resolving power similar to a resolving        power of the image set and optionally similar to within about 25        percent.

Clause 23. The microscope according to any of clauses 1 to 22 wherein:

-   -   the computational image is generated from a computational        process comprising one or more of resolution enhancement,        quality improvement, aberration correction, computational        refocusing, contrast enhancement, distortion correction, color        enhancement, registration, removing identified elements of the        data and optionally wherein the removed identified elements of        the data comprise one or more of artifact, dust or empty space.

Clause 24. The microscope according to any of clauses 1 to 23 hereinwherein:

-   -   the microscope comprises a computational microscope.

Clause 25. The microscope according to any of clauses 1 to 24 wherein:

-   -   a portion of the computational image corresponding to the area        of the sample that comprises the at least one of artifact or        empty space is generated with computational removal of dirt        related artifacts, full resolution enhancement, partial        resolution enhancement or pixel resolution enhancement without        increasing spatial resolving power of the portion and optionally        wherein the portion comprises an increased pixel count without        increasing the optical resolving power and optionally wherein        the portion is generated with interpolation.

Clause 26. The microscope according to any of clauses 1 to 25 wherein:

-   -   a portion of the computational image corresponding to the area        of the sample that comprises the at least one of the artifact or        empty space is generated without a computational process that        alters the portion image.

Clause 27. The microscope according to any of clauses 1 to 26 wherein:

-   -   the artifact comprises one or more of particulate matter,        debris, dirt, dust, or smudges.

Clause 28. The microscope according to any of clauses 1 to 27, wherein:

-   -   a portion of the computational image corresponding to the area        comprising the at least one of artifacts empty space is left        blank in the computational image and optionally in response to        skipping the area or determining not to display the area in        order to reduce data storage.

Clause 29. The microscope of any one of clauses 1 to 28 wherein:

-   -   the processor is configured with instructions to search for        valid data, and determine that the area is empty or contains        artifacts if the amount of valid data found in the area is under        a threshold amount or does not meet a defined criterion for        valid data.

While preferred embodiments of the present invention have been shown anddescribed herein, it will be obvious to those skilled in the art thatsuch embodiments are provided by way of example only. Numerousvariations, changes, and substitutions will now occur to those skilledin the art without departing from the invention. It should be understoodthat various alternatives to the embodiments of the invention describedherein may be employed in practicing the invention. It is intended thatthe following claims define the scope of the invention and that methodsand structures within the scope of these claims and their equivalents becovered thereby.

What is claimed is:
 1. A microscope, comprising: an illuminationassembly operable to illuminate a sample under observation of themicroscope; an image capture device operable to capture an initial imageset of the illuminated sample; and a processor coupled to the imagecapture device and configured with instructions to identify an area ofthe sample that comprises at least one of artifact or empty space, andto determine a process for generating a computational image of thesample in response to identifying the area; wherein the process includesa full resolution imaging for portions of the computational image thatdo not comprise the at least one of artifact or empty space.
 2. Themicroscope of claim 1, wherein the full resolution imaging includescomputational refocusing.
 3. The microscope of claim 1, wherein the fullresolution imaging includes aberration correction.
 4. The microscope ofclaim 1, wherein the portions of the computational image that do notcomprise the at least one of artifact or empty space correspond to validdata.
 5. The microscope of claim 1, wherein a higher resolving powerportion of the image comprises resolved details with a smaller spatialdistance than resolved details of a lower resolving power portion of theimage.
 6. The microscope of claim 1, wherein the computational imagecomprises a first area comprising a first resolving power and a secondarea comprising a second resolving power less than the first resolvingpower and optionally wherein the second area corresponds to the at leastone of artifact or empty space of the sample.
 7. The microscope of claim1, wherein a low resolving power portion of the computational imagecomprises an image with a resolving power similar to a resolving powerof the image set and optionally similar to within about 25 percent. 8.The microscope of claim 1, wherein the computational image is generatedfrom a computational process comprising one or more of resolutionenhancement, quality improvement, aberration correction, computationalrefocusing, contrast enhancement, distortion correction, colorenhancement, registration, removing identified elements of data andoptionally wherein the removed identified elements of the data compriseone or more of artifact, dust or empty space.
 9. The microscope of claim1, wherein the microscope comprises a computational microscope.
 10. Themicroscope of claim 1, wherein a portion of the computational imagecorresponding to the area of the sample that comprises the at least oneof artifact or empty space is generated with computational removal ofdirt related artifacts, full resolution enhancement, partial resolutionenhancement or pixel resolution enhancement without increasing spatialresolving power of the portion and optionally wherein the portioncomprises an increased pixel count without increasing optical resolvingpower and optionally wherein the portion is generated withinterpolation.
 11. The microscope of claim 1, wherein a portion of thecomputational image corresponding to the area of the sample thatcomprises the at least one of artifact or empty space is generatedwithout a computational process that alters the portion.
 12. Themicroscope of claim 1, wherein a portion of the computational imagecorresponding to the area comprising the at least one of artifact orempty space is left blank in the computational image and optionally inresponse to skipping the area or determining not to display the area inorder to reduce data storage.
 13. The microscope of claim 1, wherein theprocessor is configured with instructions to search for valid data, anddetermine that the area is empty or contains artifacts if an amount ofvalid data found in the area is under a threshold amount or does notmeet a defined criterion for valid data.
 14. The microscope of claim 1,further comprising another image capture device with a higher resolvingpower than said image capture device, the another image capture devicebeing used to image the sample at the higher resolving power than saidimage capture device.
 15. The microscope of claim 1, further comprisinga translation stage to move the sample to a plurality of locations inorder to scan the sample and generate a plurality of images, wherein themicroscope is configured to scan the sample faster in response to thearea that comprises the at least one of artifact or empty space ascompared to a scan speed of an area of the sample that comprises a validportion of the sample.
 16. The microscope of claim 1, wherein thecomputational image comprises a two-dimensional image.
 17. Themicroscope of claim 1, wherein the computational image comprises one ormore of a three-dimensional image or a three-dimensional model of thesample.
 18. The microscope of claim 1, wherein the processor isconfigured with instructions to detect the area of the sample thatcomprises the at least one of artifact or empty space, by using one ormore of: a classifier, a detail in a darkfield image, a brightness, acolor distribution, a cross correlation, a Fourier analysis, an entropy,a feature detection, a shift of details, a sparsity, an edge detection,a three-dimensional reconstruction, a depth detection, or a light field.19. The microscope of claim 1, wherein the artifact comprises one ormore of particulate matter, debris, dirt, dust, or smudges.